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(54) Service management 

(57) A system fpr collecting and aggregating data 
from network entities tor a data consuming application 
is described. The system includes a data collector layer 
to receive network flow information from the network en- 
tities and to produce records based on the information. 
The system also includes a flow aggregation layer fed 
from the data collection layer and coupled to a storage 



device. The flow aggregation layer receiving records 
produced by the data collector layer and aggregates re- 
ceived records. The system can also include an equip- 
ment interface layer coupled to the data collector layer 
and a distribution layer to obtain selected information 
stored in the storage device and to distribute the select 
information to a requesting, data consuming application. 
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Description 

BACKGROUND ' 

5 [0001] This invention relates to processes to configure and manage networks. 

[0002] Corporations and businesses are increasingly built around networks intensifying the importance of network 
reliability, predictability and performance. The network has become a mission critical component of business which 
has increased expectations for network solutions. Worldwide connectivity is not enough; users are demanding network 
services-creative, solution-oriented approaches that combine connectivity with flexible, dynamic networking options. 

w [0003] Presently a customer and a service provider can view setting up of common sen/ices with different points-of- 
view. The e.g. , an enterprise customer may be concerned with time to establish a service and ensuring that a committed 
service level is actually received. Since the enterprise customer depends on the network, anything less than instanta- 
neous attention to network failure might be considered unacceptable. The service provider is concerned with keeping 
up with the volume of service requests. Providers must efficiently deliver services and offer new competitive services 

15 while ensuring quality delivery and accurate billing at a manageable, reduced cost. 

[0004] The infrastructure in place to satisfy service requests of today is largely manual. The workflow necessary to 
turn a service request into service delivery typically passes through many different people via faxes, e-mail, and paper 
memos and so forth. Manual processes are error-prone and include many delays that lengthen service delivery. 
[0005] There are custom scripts, spreadsheets for billing, and isolated applications for diagnosing problems. Most 

20 of these tools do not share data and often duplicate data. Providing customers with reports of their network use is 
difficult since these reports tend to span data from many tools. 

SUMMARY 

, 25 [0006] According to an aspect of the invention, a computer implemented method for managing network service in- 
cludes receiving, from a subscriber, a request for a selected network service and determining from the request for a 
selected service configuration characteristics for resources. The method also includes configuring the resources with 
configuration characteristics to provide the selected service and monitoring the service provided. 
[0007] According to an additional aspect of the invention, a system includes a policy enabled network server that 

30 can decompose a request for a level of service for the network and produce a template that defines the service and a 
service provisioning process to produce a network configuration from the template that defines the service. The'system 
also includes an accounting process that monitors the level of service and produces network accounting records, 
[0008] The summary of the invention does not necessarily disclose all the features essential for defining the invention; 
the invention may reside in a sub-combination of the disclosed features. 

35 [0009] One or more of the following advantages may be provided by one or more aspects of the invention. 

[001 0] The invention addresses network growth by simplifying management, managing the network as a system and 
providing service-level management. The invention is not limited to management of an particular service. Rather the 
invention provides an integrated management solution for new services (and yet-to-be-defined services). Networks 
are not viewed a collection of largely static protocols. Rather, networks are dynamic, constantly adapting systems that 

40 require extensible, simplifying management solutions. These features address the integration of network services pro- 
viding "IP-optimized networks" necessary to satisfy demands for networking. 

[0011] The invention addresses these deficiencies by providing an automated, integrated system that is targeted at 
covering the entire life cycle for creating, delivering, and billing services. When workflows are integrated, they are faster 
and less error-prone. In fact, automated workflows provides a new level of service-services that can dynamically 
•*5 change, perhaps directly changed by the end customer via a customer network management interface into the service 
management system. Since services are integrated, data is easily shared across ail processes that are part of the 
service life cycle. 

BRIEF DESCRIPTION OF THE DRAWINGS 

50 

[0012] The following description is of a preferred embodiment by way of example only and without limitation to the 
combination of features necessary lor carrying the invention into effect. 

[0013] FIG. 1 is a block diagram of a server running an accounting application monitoring a network: 
[0014] FIG. 2 is an architectural block diagram of the accounting application used in FIG. 1 . 
55 [0015] FIG. 3 is a block diagram of accounting support in an access process used by an Internet/Intranet service 
provider or a large enterprise. 

[0016] FIG. 4 is a block diagram of accounting support in an access process used by an Internet/Intranet service 
provider or a large enterprise using an Extranet switch. 
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EXEMPLARY CONFIGURATIONS 

[0049] Referring hbw to FIG. 3, the accounting process 14 can. in general/support any configuration: Exemplary 
configurations used by an Internet service provider 100, an Enterprise A that host its own remote access 110, and an 

5 Enterprise B that uses the Internet service provider 120, are shown. 

[0050] As shown in FIG. 3, for the Internet service provider, data collectors 52a-52d are distributed at specific Points 
of Presence (POP), such as remote access concentrators 102 managed by the Internet service provider. The remote 
access concentrators allow, a mobile user 106 or an Internet user 107 with remote access to access an enterprise 
over the Internet, via the Internet service provider. In this example the Internet service provider arrangement 100 and 

70 the large Enterprise arrangements 110 and 120 include servers 13, 1 3', and 13" that run accounting processes 14, 14' 
and 14' The accounting processes 14, 14' and 14" each independently manage and collect information regarding 
network traffic usage. 

[0051] The Internet service provider arrangement 100 includes the accounting server 13 that runs the accounting 
process 14. The accounting process 14 includes a flow data collector layer 18 that gathers data from the service 

is provider network 1 00. The flow data collector layer 1 8 includes distributed, individual flow data collectors 52a-52d. The 
distributed, flow data collectors 52a-52d collect transaction specific details about a user's connection type and actual 
network usage. These data are converted into the NARs in the distributed, flow data collectors 52a-52d, as mentioned . 
above. The NARs are aggregated over the entire system by the flow aggregation layer 60 (FIG. 2). 
[0052] Data is made available to the Internet service provider via the data distribution layer (FIG. 2) so that the 

20 Interne! service provider can analyse data in order to differentiate service offerings to different users. The Internet 
service provider can evaluate and use such detailed accounting data collected from various sources to migrate from 
a single flat rate fee business model to more flexible charging. For example, analysis of the data can enable the Internet 
service provider to develop new service options that can take into consideration bandwidth usage, time of day, appli- 
cation usage and so forth. In addition, Internet service providers can offer discounts for web hits that may exist in an 

2S Internet service provider cache, thereby minimizing the need to look up an address for a requested site on the Internet 
and can provide free E-mail usage while charging for other types of applications such as file transfer protocol and web 
traffic. 

[0053] The data can also be used by other applications such as network planning, security, auditing, simulation, flow 
profiling capacity planning and network design and so forth. Thus, the Internet service provider can independently 

30 monitor and evaluate network traffic caused by remote employees and mobile users, for example. 

[0054] Similarly, other instances 14', 14" of the accounting process can be used by enterprises, as also shown in 
FIG. 3. For example, an enterprise may host its own remote access," as shown for Enterprise A and would include a 
server 13' running an accounting process 14V An enterprise could use the Internet service provider as shown for 
Enterprise B, and still have a server 1 3" running an accounting process 14". The accounting process 14', 14" includes 

35 an associated data collector that is coupled to enterprise A and enterprise B local area networks or other network 
arrangement. In this model, the enterprises use data from the accounting process 14\ 14" for enterprise charge-back 
functions such as billing departments for Internet usage within the enterprise and so forth. 

[0055] Different instances of the accounting process are used by both the Internet service provider and enterprise 
A and Enterprise B sites The instances 14, 14", 14" of the accounting process are independent they do not need to 

40 exchange accounting data Rather they exist as separate, independent accounting domains. 

[0056] Referring now to FIG 4 a similar access configuration 100', as the configuration 100 (FIG. 3) can be used 
with an Extranet switch 122 Extranet access allows remote users to dial into an Internet service provider (ISP) and 
reach a corporate or branch office via an ISP. The Extranet switch allows Internet users access to corporate databases, 
mail servers and tile servers lor example. It is an extension of the Internet in combination with a corporate Intranet. In 

■« this configuration, the Extranet switch 122 can be owned and operated by an Internet service provider as shown with 
enterprise A, or it could,. alternatively, be owned and operated by an enterprise, as shown with enterprise B. Users 
would access the corporate network of either enterprise A or enterprise B, via the Internet service provider with various 
types of tunneling protocols such as L2TR L2F, PPTPor IPSec : and so forth. The accounting server 1 3 located at the 
service provider and also accounting servers 13', 13" within enterprise A and enterprise B allow each the Internet 

so service provider and each of enterprises A and B to run accounting process 14', 14" on the servers 13', 1 3" to monitor 
and collect network data. 

TRANSACTION FLOW MODEL 

55 [0057] Referring now to FIG 5. a graph 140 depiction of a very large scale network includes a device "A" 142 com- 
municating with a device "B" 144. The graph 140 includes nodes (not all numbered) that can represent routers, switches, 
flow probes; etc. that have interfaces (not shown) which maintain statistics on information passed through the interfaces. 
For example, a switch may have a number of Ethernet ports and a host could be connected to one of the ports and in 
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communication with one of the interfaces to transfer information over the network. The interface would have counters 
that are used to track "packet's in, "packet's out, "bytes in, bytes out", and so forth. 

[0058] In this case of the host connected to the port, or a router or some other device being connected to the port, 
there is no other connection that the host, router or other device is aware of other than the entire network. This is an 
example of a "connectless oriented" protocol. A data collector 52 can be disposed in the network in a path between 
the entities "A" and "B", such that the data collector 52 monitors some of the packets that comprise a flow between "A" 
and "B." As a single point monitor, the data collector has no concept that there are two ends communicating. The data 
collector 52 identifies these entities "A" arid "B" in various NARs produced by the data collector 52. At later stage in 
the processing, either in the data collector 52 or elsewhere in the accounting process 14 the NARs are correlated so 
that the NARs or some aggregated NAR produced by the data collector 52 or the rest of the accounting process 14 
can be associated with the accountable entities "A" and "B" to thus identify a connection between entities "A" and "B." 
[0059] The data collectors 52a-52g (FIG. 2) develop a description of the connection. For a router, normally an address 
of the object that is connected to that interface might serve as an address. A switch has an IP address that can be 
used as the destination. The actual device that is connected to the switch or router can be used as an accountable 
object. Although the traffic is not destined for the router, the data collector can use those identifiers as keys to the 
endpoints "A", "B." 

[0060] In many cases, the protocol can explicitly determine connections. For example, the TCP/IP protocol is explicitly 
a "connection oriented" protocol used in the Internet. When the data collector 52 needs to determine a connection, the 
data collector 52 can exploit the "connection oriented" nature of certain types of protocols such as the TCP/IP protocol. 
When the data collector 52 tracks a TCP/IP connection, the data collector 52 can determine exactly that A and B are 
connected, when the connection starts, stops, and updates. With other protocols such as a "connectionless" protocol, 
and even in some complex environments such as a virtual private network or a proxy server, the data collector 52 does 
not necessarily know the real endpoints. The data collector 52 only knows that some entity is talking to some other entity. 
[0061] Thus, the data collector 52 is a single point monitor, that monitors traffic at one point in the network and 
converts the traffic into a "pipe oriented" or "flow oriented" accounting information. The data collector 52 identifies a 
source and a destination of the traffic. That is, the data collector develops a "connection oriented tracking." By distrib- 
uting data collectors 52a-52g (FIG. 2) through out the network the network can be modeled as pipes having two end- 
points. A data collector can be disposed in a partial pipe. The data collector 52 determines that one end of the pipe 
refers to "A" and the end of the pipe refers to "B." The data collector 52 can be disposed anywhere along the network. 
[0062] The graph 1 40 represents the network as a directed graph, including partial segments! The endpoints of those 
partial segments can act as proxy entities to the actual accountable objects. Once independent accounting records 
that relate to these two entities A and B are aggregated in the accounting process 14, the accounting process 14 can 
identify that A and B are connected and have particular metrics. 

[0063] Some equipment have a half pipe model that generate independent accounting records for each half pipe. 
The data collectors can assemble full pipe information from half pipe information. The accounting process could be 
coupled to equipment that gives a half pipe model for A communicating with B and a separate one for B communicating 
with A. The data collectors 52a-52g combine information from these two half pipes into a bidirectional flow. 
[0064] Referring now to FIG. 6, an example of data flow 130 through the accounting process 14 is shown. In this 
example, data flow is initiated by a user 1 31 making a call to a remote access concentrator (RAC) 1 32. Upon receiving 
the call, the RAC 132 authenticates the user against a secure access controller 134. After verification, the RAC 132 • 
connects the user to the network 135 and sends a RADIUS Start record (not shown) to the accounting process 14. 
The accounting process 14 generates a RADIUS Start NAR 1 37a and stores the RADIUS start NAR in a database 62. 
At that point, the remote user may check e-mail, look at a web server and transfer a file. For each transaction, the 
accounting process 1 4 captures the IP traffic, generating a e-mail, http, and ftp network accounting records 137b-1 37d, 
respectively These are stored in the database 62. Upon completion of these transactions the user would log out of 
the network, at which time the RAC would send the accounting process 14 a RADIUS Stop record. The accounting 
process 14 generates a RADIUS Stop NAR 137e and stores the RADIUS stop NAR in the database 62. All of these 
records reflecting Ihe user's transactions could be viewed and reported in flexible ways dependent on the needs of an 
end-user application. 

NETWORK ACCOUNTING RECORDS (NARs) 

[0065] The data collector 52 translates collected information into network accounting records (NARs). A NAR rncfucfes 
a header, an accounting entity identifier and metrics. The network accounting record (NAR) is a normalized data record 
that contains information about a user's network usage activity. 

[0066] Referring now to FIG. 7 a base level "activity" NAR that includes source, destination, protocol, source port, 
destination port, byte and packet counts, etc The base level activity NAR can be combined and aggregated in many 
different and flexible ways to support various accounting functions. The NAR is an activity record corresponding to 
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some level of detail. Detail can vary based on the level of aggregation being applied, in accordance with the needs of 
the end-user/application. 

[0067] FIG. 7 has at one level 1 52 a plurality of exclusively "Activity NARs" which could correspond to a very low 

level of detail, or could be the result of a prior aggregation providing a higher level view of the information. Thus, FIG. 
5 7 shows a collection 152 of exclusively activity NARs. From base level data, additional "views", of the NAR could be 

produced, such as a set of "Summary NARs" 1 54, or another set of Activity NARs 1 56 which could be a result of further 

aggregation of the base level information, or lastly a combination of a set of Summary NARs and Activity NARs 158. 

The summary NAR is produced by the central aggregation layer 60 andean include user identifying information, protocol 

information, connection time information, and data information, and so forth. 
10 [0068] The specifics of what can be combined and aggregated will described below. Thus, the accounting process 

14 use of NARs provides the ability to combine and aggregate base level activity information in a flexible way to meet 

the specific needs of the end-user/application. 

[0069] TABLE 1 below corresponds to the fields that can be captured in a NAR. This is essentially the activity NAR. 
The NAR contains these fields, which can then be combined and used to form other activity NARs or summary NARs. 

75 



TABLE 1 



Column Name 


Description 


OSA_SOURCE_TYPE 


List all of the possible data sources frorn which data can be 




COM6C16Q. ncl cl fcJI lUfcJ IU UOn_OUUnOC 1 T i i_ 1 nDLC . 


OSA_SOURCE_SERIAL_NUM 


Number which uniquely identifies an OSA FDC. 


START_TIME_SEC 


Indicates the date and time at which a record was recorded. 


START_TIME_USEC 


Microseconds component of START_TIME_SEC. 


SEQUENCEJMUMBER 


Sequence number assigned by the source of the NAR to 




uniquely identify the record and ensure integrity. 


USER.NAME 


The user associated with the record. 


EVENT 


Event type of the record (i.e. Start, Stop, Update). 


SESSION J D 


Unique Accounting ID to make it easy to match start and stop 




records. 


SESSION_TIME 


Duration of the record in seconds. 


SESSIONJTIMEJUSEC 


Microseconds component of the SESSION_TIME. 


SRC_ADDR 


Source address of the record. 


DST_ADDR 


Destination address of the record 


PROTO 


Protocol of the record. Reference to 




UoA_rnU 1 UUUL_ 1 Yrt laDie. 


SRC_PORT 


Source port number. 


DST_PORT 


Destination port number. 


SRC_OCTETS 


Number of bytes transmitted into the network by the source. 




• For RADIUS is equivalent to Acct-lnput-Octets. 


DST.OCTETS 


Number of bytes sent out of the network, to the source. For 




RADIUS is equivalent to Acct -Output-Octets. 


SRC.PKTS 


Number of packets transmitted into the. network by'the 




source. For RADIUS is equivalent to Acct-lnput-Packets. 


DST_PKTS 


Number of packets transmitted out of the network, to the 




source. For RADIUS is equivalent to Acct -Output-Packets. 


SRC.TOS 


The Type of Service coding marked by the source. 


DST_TOS 


The Type of Service coding marked by the destination. 


SRC.TTL 


The Time To Live field set by the source and modified by the 




network until the Nortel flow probe recorded it. 


DST_TTL 


The Time To Live field set by the destination and modified 




• by the network until the Nortel flow probe recorded rt. 


OSA_CAUSE 


A number that indicates the reason the accounting record 




was generated. 


OSA_STATUS 


A value used to indicate the status of an accounting record 




when it was created. 
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TABLE 1 (continued) 



Column Name , ' 


Description 


ACCT_DE LAY_TI ME ( 

AC CT_ AUTHENTIC 
ACCT_TERMINATE_CAUSE 
ACCT_MULTi_SESSION_ID ACCT_LINK_COUNT 


Indicates how many seconds the client has been trying to 
send this record 

Indicates how the user was authenticated. 

Indicates how the session was terminated 

Unique Accounting ID to make it easy to link Indicates the 

count of links which are known to have been in a given 

multilink session at the time the accounting record is 

generated. 



[0070] The summary NAR and activity NAR have a one-to-many relationship. That is, while there can be a single 
summary NAR for a particular user over a particular call that will contain information about the sum of usage of network 
resources over the duration of the call, there can be many activity NARs. The activity NARs capture details about the 
actual activity and applications being used during the call. The summary NAR, therefore, depicts the total expense of 
the transaction or a set of transactions on a network, whereas, the activity NARs depict expenses of a transaction at 
any point in time. The summary NAR is generated in the flow aggregation process 60, as will be described below. In 
essence, the summary NAR is generated from individual activity NARs correlated in the data collectors 52a-52g, as 
will be described below 

[0071] A NAR is a member of a generic data message set that is used to transport data, such as network Accounting 
date, through the accounting process 14. These system data messages include "Status Event - , "Maintenance Event", 
"Trace Event", "Network Accounting Record". Accounting process 1 4 messages share a common MSGJHDR structure 
that is used to discriminate between the four types of accounting process 14 messages. The Message Header 
(MSGJHDR) includes Message Type, an Message Event and Cause, and Message Length. 

NETWORK ACCOUNTING RECORD DATA STRUCTURES 

[0072] As will be described below, the NAR is unique within the accounting process 14. The NAR has a NARJD that 
specifies an accounting process component ID. The component ID specifies the data collector assigned to a particular 
network device that produced the NAR. The component ID e.g., NAR_SRCJD 203a (FIG. 8B below) is allocated at 
the time that the component is produced. The NARJD also includes a time stamp at the second and microsecond 
level so that the accounting process 1 4 can discriminate between multiple NARs generated by a particular component. 
A sequence number, e g.. 32 bits is also used. to differentiate NARs from the same accounting process component 
that may have the same time stamp. The sequence number e.g., NAR_SEQ_NUM 203c (FIG. 8B) is preferably a 
monotonically increasing number starting from, e.g., 1 . As long as the component is functioning, and producing NARs, 
the component provides a new sequence number to a new NAR. 

[0073] Referring now to FIGS 5A-8C, a Network Accounting Record (NAR) data structure 200 is shown. 
[0074] As shown in FIG 8A the NAR data structure 200 includes two basic accounting objects, a Network Accounting 
Record Identifier 202, and optionally one or as shown a plurality of Network Accounting Record Attributes 204a-204n, 
generally denoted as 204 The Network Accounting Record Identifier 202 has a set of object identifiers that uniquely 
identifies the network accounting record within the accounting process 14. 

[0075] The Network Accounting Record Identifier 202 acts as a database key value that makes the NAR 200 unique 
within the entire accounting process 14. The Network Accounting Record Identifier 202 allows the NARs to be handled 
and managed using database functions such as database integrity analysis and reliability analysis. The Network Ac- 
counting Record Identifier 202 also gives the accounting process 14 the ability to track the source of NARs and to build 
mechanisms such that the accounting process 14 can maintain identity of the origination of NARs throughout the 
system 10. 

[0076] The plurality of Network Accounting Record Attributes 204a-204n provide metrics for the NAR 200. The Net- 
work Accounting Record Attributes 204a-204n capture specific information contained in data from network devices. 
Differentiating between the entity identifier 202 and the metric 204 allows the accounting process 14 to perform logical 
and arithmetical operations on metrics 204 while leaving the accounting identifier intact 202. The accounting identifier 
202 can be enhanced unlike the metrics. 

[0077] The data collectors 52a-52g (FIG. 2) are oriented around the process of filling in the NAR. The metrics are 
left untouched by the data collector and are passed transparently into the accounting process flow aggregation process 
60. The data collectors 52a-52g assign the accounting entity identifiers 202 to the metrics e.g., a source and a desti- 
nation. identifier to the metric. In the example of a router link, the metrics that the router interface provides are in the 
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form of "information in" and "information out" e.g., octets in, octets out, bytes in, bytes out datagrams in, datagrams 
out, faults in faults out and so forth. The data collectors 52a-52g determine what "in" and "out" means and assigns 
the unique identifier that is unambiguous relative to the determined meaning of "in" and ■out." Once a data collector 
52 has established this convention, the convention is used throughout the system 10. 
5 [0078] Thus the NAR Identifier 202 provides database constructs to a NAR, whereas, the plurality of Network Ac- 
counting Record Attributes 204a-204n provide the actual metrics used for network activity reporting and network ac- 
counting 

[0079] As shown n FIG BB ! the Network Accounting Record Identifier 202 (NARJD) is a set of objects within the 
NAR thnt miquci/ *ccn:i1ies the NAR throughout the accounting process 14. The NARJD 202 is designed to support 

to- a numbci oi properties ol n NAR including flexibility, accountability, reliability and uniqueness. In order to provide these 
properties if»c NAR ID 202 is divided into objects designed to specifically provide these properties. Flexibility is sup- 
ported throjqh n NAR HCR 203 section of the NARJD. Accountability is attained in the NAR through explicit identi- 
fication of nc source o! mc NAR by a component identification NAR_SRC J D 203a. The source time is maintained in 
a NAR SRC TIME 203b Reliability is supported, as described above, through the use of a NAR sequence number 

is (NAR^SEQ NUM i 203c which is designed to enable traditional database integrity mechanisms. 

[0080] I he NAM U 202 is tscd to provide uniqueness for each NAR. The responsibility for guaranteeing the unique- 
ness ol CHcn NAM is handled by every accounting process component that has the ability to originate/source network 
accounting records This responsibility requires that each accounting process component have the ability to unambig- 
uously identify itscl! in each NAR that it produces. Thus, NAR type identifier, NAR_TYPE, is comprised of the source 

20 . component identifier NAR SRCJD. the NAR source lime, NAR_SRC_TIME, and the NAR sequence number, 
NAR_SEQ_NUM Thubc three data objects act as a database key for a particular network activity record, ensuring the 
uniqueness of the NAR throughout the entire system. , 

[0081] The NAR_SEG_NUM can have several purposes. One way that the NAR_SEQ_NUM can be used is as a 
discriminator when two NARs are produced at the same time. A second way that the N AR_S EQ_N U M is used is as a 
25 monotonically increasing index to ensure database integrity Because the NARJD is unique, it should be considered 
as an allocated value A NARJD is allocated at NAR origination. 

[0082] If a component creates or modifies the contents of an existing NAR, as for example when aggregating two 
NARs together, the component originates the NARJD. This provides an opportunity for the accounting process 14 to 
have explicit internal integrity mechanisms that can account for any network accounting record that is processed by 
30 the accounling process 14 The NAR Source Identifier NAR_SRC_ID 203a includes a source type 207a and a Source 
Serial Number 207b. The serial number 207b is an administratively allocated value e.g., 24-bits that uniquely identifies 
the NAR source type throughout the accounting process 14. The source serial number 207b should be unique within 
the specific accounting domain 

[0083] The (NAR_SEQ_NUM) 203c is a monotonically increasing, e.g., unsigned 32-bit integer that acts as a se- 
35 quence number for NARs that originate from a particular NAR source. Because the value of the NAR_SEQ_NUM can 
"wrap around", the combined 64-bit value NAR_SRC JD and NAR_SEQ_NUM are unique only over a specified time 
period. 

[0084] Referring now to FIGS 9A-9B. exemplary formats for Network Accounting Record Attributes 204 are shown. 
There are two variations on a single NAR_ATTR I BUTE format that can be used. As shown in FIG. 9A, a standard 
40 NAR_ ATTRIBUTE formal 206h includes header fields NAR_ATTR type, NAR_ATTR Code, NAR_ATTR Qualifier and 
NAR_ATTR Length and a "value held ' In order to conserve the size of accounting information, when the size of the 
value of the NAR_ATTRIBUTE is a byte i e. ; 8-bits ..as indicated in the NAR -ATT R Qualifier field, the format 206b of 
the NAR_ATTRIBUTE can be as shown in FIG 9B ; including fields NAR_ATTR type, NAR_ATTR Code and an 8-bit 
NAR_value field. 

45 [0085] Each supported object is assigned an NAR_ATTR Code. Through the NAR_ATTR Code, the accounting proc- 
ess 14 can distinguish the semantics of a particular NAR_ ATTRIBUTE. Although NAR_ATTR Codes are specific to 
the NAR_ATTR Type, the NAR^ATTR Code assignments can be unique to aid in implementation. Values can be as- 
signed to piovide some explicit hieiarchical sir ucluie. Each NAR_ATTR has an 8-bit NAR_ATTR Qualifier that provides 
typing infoimation foi the NAR_ATTR. The NAR_ATTR Qualifier is used because some supported objects can be 

50 represented using several different types. Counters, for instance can be 32-bit as well as 64-bit, in the case of aggre- 
gated objects. Network identifiers may use numeric indexes, or strings as labels. The NAR_ATTR field specifies the 
length of the NAR attribute including the N AR_ATTR header. 

[0086J There arc five types of Network Accounting Record Attributes that are supported rn the NAR. Thefive attributes 
are Accounting Time Interval (ACCT_TIME) (FIG. 10): Accounting Entity Identifier (ACCT_ENTITYJD), (FIGS. 11 A- 
55 HE): Accountable Entity Descriptor (ACCT_ENTITYJDesc); Network Activity Metrics (NET_METRICS)(FIG. 12); and 
two Transparent Attributes (TRANS_ATTR)(FIGS. 13A-13B). As necessary, additional NAR_ATTRIBUTES can be sup- 
ported For example, a NAR_ ATTRIBUTE type could also include Security Attributes for accounting data to protect 
against unauthorized introduction or modification of accounting information. 
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[0087] Referring now to FIG. 10, an Accounting Time Interval record includes a value "seconds" and a value "micro 
second". The values of ■seconds" and "micro seconds" together represent a time stamp of network activity for the N AR, 
as discussed above. When derived from an absolute time value that represents the end of the accounting time interval, 
the Accounting Time Interval is the duration, as calculated using the Accounting Time Interval as the starting time value. 

5 All Network Accounting Records can have an Accounting Time Interval attribute. 

[0088] Referring now to FIGS: 11 A-11E, Accountable Entity Identifier data structures are shown. The Accountable 
Entity Identifiers are a collection of entity description attributes that together identify an accountable entity in the ac- 
counting process 14. The accounting entity identification mechanism facilitates flexible NAR aggregation properties of 
the accounting process 14. The ACCT_ENTITYJD is the description of an accounting object within the accounting 

10 process 14. There can be one or more ACCT_ENTITYJDs in a given NAR; but there must be at least one 
ACCT_ENTITY_ID in an Network Accounting Record. The actual accountable object is defined by the entire collection 
of ACCT_ENTITYJDs that are included in the NAR. 

[0089] In transaction based accounting, a network accounting record will contain two ACCT_ENTITYJDs, repre- 
senting the source and the destination entities that are involved in the network transactbn. For traditional flow based 
is accounting, these would normally be the two network addresses that are involved in the flow. Qualifiers are available 
in the ACCT_ENTITYJD objects to indicate which ID is the source and which is the destination of the network trans- 
action. 

[0090] In direct support of flow based accounting data sources, the accounting process 14 supports a specific IP 
flow descriptor. This is the traditional I P 5-tuple f low description . The accounting process 1 4 could also support a 6-tuple 
20 flow descriptor that includes a type of service (TOS) indicator in the flow designator. This allows for Glass of Service 
distinction in the accounting model. 

[0091] For network activity data sources that do not have a transaction accounting model, there may only be a single 
ACCT_ENTITY_ID present in the accounting record. Qualifiers for the ACCT_ENTITYJD are available to indicate if 
the single object is the source, destination, or both, for the accounting metrics that will be included. The types of entities 
25 include User Identifiers and Network Entity Identifiers. The network identifiers can include IP Address, Flow Description, 
and Network Object ID. Other types of accounting entities can be provided. 

[0092] The actual accountable entities for a specific network accounting record are specified in the complete set of 
ACCT_ENTITYJD(s) that are present in the NAR. 

[0093] Operations that can be applied to N ARs, specifically aggregation, can influence how ACCT_ENTITY J Ds are 
30 used in N ARs. Each accountable entity identifier that is present adds refinement to the definition of what accountable 
entity the metrics actually apply to, whereas each ACCT_ENTITY_DESC further refines the description of the account- 
able entity. 

[0094] Referring now to FIG. 11 A, a NARJJSERNAME is a specific type of NARJJSERID data structure. A system 
string type "Username' 1 222 can represents a real accountable user within the accounting process 14. The 

35 NARJJSERNAME data structure 220 is used to transmit the string. The semantics can be applied when the string 
"Username- 222 is supplied by RADIUS or from DCHP management systems. The NAR_USERNAME data structure 
220 includes a NARJJSERNAME NAR_ATTR Qualifier that provides for Role designation, indicating whether the 
object referenced is acting as a source, destination, both or undeterminable within the system. The N AR_ATTR Qualifier 
bits are set when the Role can be determined without ambiguity. 

40 [0095] Referring now to FIG. 1 1 B, a N AR_USER JD data structure 230 is the general type for identifying an account- 
able user. The accounting process 14 can use any available object type to represent the NAR_USERJD value 232. 
The NAR_USERJD value 232 will be a system established STRING type or a user index as generally supplied by a 
database system: The semantics of the NARJJSERJD value 232 are consistent within the accounting process 14, 
and can be consistent outside of the accounting process 14. 

45 [0096] Referring now to FIG. 11 C, a NAR JP_ADDRESS data structure 240 is shown and which is the general net- 
work component identifier for an IP enterprise network. NAR J P_AD DRESS data structure 240 includes a IP Address 
242 that is usually unique within the accountable domain, and thus can be usable as an accounting process 14 identifier 
Within the accounting process 1 4, the occurrence ol this record implies that the address is unique within the accounting 
realm. NARJP_ADDRESS type includes a NAR J P_ADDRESS NAR„ATTR Qualifier. The NARJP_ADDRESS 

50 N AR_ATTR Qualifier provides for Role designation, indicating whether the object referenced is acting as a source, 
destination, both or undeterminable within the system. These bits are set when the Role can be determined without 
ambiguity. 

[0097] Referring now to FIG . 1 1 D, a N AR_NETWORK J D type data structure 250 rs shown. The hfAR_NETWORfCJD 
data structure 250 includes a NETWORK JD value 252 is a general type used for identifying a network component 
55 when a network address is inappropriate. The accounting process 1 4 can use any available object type to represent . 
the NARJslETWORKJD, but it is assumed that this value will be an accounting process 1 4 established STRING type, 
(e.g., a Media Access Control (MAC) address that is predefined in Network interface cards), object type or a number 
index that cannot be associated with a network address. The semantics of the NARJslETWORKJD is consistent within 
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the accounting process 14, and can be consistent outside the accounting process 14. A N AR_N ETWORKJ D 
NAR_ATTR Qualifier provides for Role designation, indicating whether the object referenced is acting as a source, 
destination, both or undeterminable within the system. These bits are set when the Role can be determined without 
ambiguity. 

[0098] Referring now to FIG. 11 E, a NAR_FLOW_DESC data structure 260 is the general type for reporting on flow 
based network activity. The NAR_FLOW_DESC is a composite data structure 260 including a IP Source Address 262, 
IP Destination Address 263, Transport Protocol 264, Type of Service 265, Source Port 266 and Destination Port 267 
that are populated from transport and network layer of IP packets via flow probe. The NAR.FLOWJDESC NAR_ATTR 
Qualifier provides for Role designation, indicating whether the object referenced is acting as a source, destination, 
both or undeterminable within the system. These bits are set when the Role can be determined without ambiguity 
[0099] Therefore the Network Accounting Activity Records are fundamentally bindings between an accountable entity 
and a set of metrics that can be associated with that entity over a specified period of time. The NARs provide flexibility 
in defining, or specifying, the accountable entity. This level of flexibility is required because in network accounting, an 
accountable entity could potentially refer to objects that are either physical or logical, singular or members of collections, 
or geographically or topologically constrained, such as network numbers or autonomous system numbers. 
[0100] A set of accountable entities includes Username and Network Object Identifiers. There can be additional 
descriptive information available within network activity reports and within networking components that could be used 
to further describe accountable entities. These entity attribute descriptors can be used in the accounting process 14 
to provide additional flexibility in how network activity information is reported and tallied. Support for entity descriptions 
can include object support for: 

Flow Descriptors 

Flow Protocol Descriptors 
Flow Transport Port Descriptors 

Authentication Descriptors 

NAS Descriptors 
Aggregate Descriptors 

Class Identifiers 
Session Identifiers 
Multi-Session Identifiers 
VLAN Identifiers 
ELAN Identifiers 
Group Identifiers 

Access Identifiers 

Source and Destination Ethernet Addresses 
Ingress and Egress Tunnel Ids 
Ingress and Egress Port Numbers 

ATM Virtual Circuit VPI/VCI 
Calling and Called Station Ids 

Flow Status Descriptors 

Class of Service Identifiers 
Quality of Service Identifiers 
Traffic Path Identifiers 

Accounting Time Interval 
Accountable Network Activity Metrics 

Source and Destination Datagrams 
Source and Destination Octets 
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Extended Network Activity Attributes 

Network Flow Control Indications 
Host Flow Control Indications 
5 Traffic Burst Descriptors 

[0101] Rclcmnq now to FIG. 12, a NETJvlETRIC data structure 270 is shown. A NEt_METRIC data structure 270 
to suppon h count is shown in FIG. 14. The NETJvlETRIC data structure 270 is used to hold basic accounting values 
that can be tn hed withn the accounting process 14. The NET_METRIC data structure 270 can support time, octets, 

10 datagram counts nrd cells circuits, tunnels and so forth. 

[0102] Reicmnq now to FIGS. 13A and 13B, two basic transparent objects TRANS_ATTR objects are shown; UN- 
DEFINED 2r0 nrti RADIUS 290 New TRANS_ATTR object types can be defined as needed. These are objects that 
a user mny wnnt to send throjgh the accounting process '14, that are customer specific, or proprietary in nature. The 
account ng process 14 nliows for object transparency, i.e., an object that the system does not act on or modify. Thus, 

is the contents ol transparent attributes are undefined with respect to the accounting system. They are passed through, 
unmodified 

FLOW DATA COLLECTOR , 

20 [0103] Rolemncj to FIG 14 h (tow data collector system 300 for supporting the flow data collector ("FDC") 52 (from 
FIG. 2) is shewn' Ttiu flow Untrt collector system 300 includes a processor 302 coupled to a'memory 304. In this 
embodiment the FDC is a process stored in the memory 304 and executed by the processor 302. The FDC 52 includes 
several NAR processing components or processes. These processes include a NAR constructor 306 for converting 
data gathered by the equipment interface (El) 1 6 (shown in dashed lines) from a network device or technology ("network 

2S entity tt )into NAR format Recoil that each equipment interface 42a-42g is associated with an flow data collector. Thus, 
the combination of a equipment interface and a flow data collector support a particular device or technology and collects 
data from the particular device or technology using a pre-defined format, schedule and protocol specific to that device/ 
: technology. The NAR processes further include a correlator 308, an enhancement process 31 0 and an aggregator 31 2 
for processing the constructed NARs as appropriate The details of these processes will be discussed further with 

30 reference to FIG 15 below 

[0104] Still referring to FIG 14. the memory includes a local store 314 and a flow data collector configuration (file) 
318. The local store 31 4 stores data received from the equipment interface 16 and processed NARs. The configuration 
file 318 is provided at startup to configure the flow data collector 52. The configuration file 318 specifies various con- 
figuration parameters 319 including a time parameter 320 and a policy 322. The NAR processes 304 populate and 

35 process NARs for data received from network devices via the equipment interface 16 in accordance with the policy 
322 of the configuration file NARs being held in the local store 31 4 are transferred to the flow aggregation process 60 
(FIG. 2, shown here in dashed lines) when the time specified by the time parameter 320 expires. 
[01 OS] It can be appreciated from the above description that the flow data collector 52 is a software component of 
the accounting process and runs on the flow data collector system 300. The flow data collector system may be any 

•*o computer system, such as a workstation or host computer, which can communicate with the equipment interface. 
Alternatively, the FDC may reside in the network devie'e itself Many known details of the flow data collector system 
300 have been omitted from FIG 17 lor the sake of clarity, as the figure is intended to highlight the processes of and 
memory structures associated with the flow data collector. 

[0106] Conceptually, as earlier described, each flow data collector of the accounting process architecture is capable 
•*5 of supporting multiple equipment interfaces 16. At the implementation level, there is a one-to-one correspondence 
between each flow data collector "process" and a given equipment interface 1 6. For example, a single computer system 
might provide both RADIUS and flow probe support and thus run separate flow data collector processes for the RADI US 
Eland the flow probe equipment interface. In such a configuration, where the flow data collector processes are operating 
independently and loading directly into the flow aggregation processor 60 (FIG. 2), the computer system itself may be 
so viewed as an flow data collector supporting multiple Els. 

[0107] Referring now to FIG 15, a data collection process 3X performed by the flow data collector 52 of FIG. 17 is 
shown. The flow data collector receives 332 data from the equipment interface for an network device. The flow data 
collector performs an equipment interface specific translation to convert 336 the received data into WAR format as weff 
as populates the NAR header. Once the NAR is populated with the appropriate data, the flow data collector 52 attempts 
55 to correlate 338 the newly populated NAR with other NARs. That is, the flow data collector 52 compares the newly 
populated NAR to NARs currently stored in the local store 31 4 (from FIG . 1 4) to determine if there are multiple instances 
of the same object. Specifically correlation is performed by examining the ACCT_ENTITY_ID (from FIGS. 11A-11E). 
[0108] The flow data collector uses one clock and one time determinator, so all NARs that the flow data collector is 
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processing or holding are assumed to be in the same time domain. Consequently, the flow data collector need not 
consider time during correlation. If the flow data collector 52 determines that a NAR ACCT_ENTITY_ID (i.e., the col- 
lection of descriptors or objects as described above) in the NAR matches that of another NAR that it is currently holding, 
the FDC 52 can replace an older (stored) NAR with the new (i.e., most recently populated) NAR and discard the older 

5 NAR. For example, the existing or older NAR may be a start record and the new NAR a stop record that includes all 
the data included in the older NAR, thus superseding the older NAR. Alternatively, if the new NAR is a replica of an 
existing NAR, the FDC may decide to discard the new NAR. Also, the data collector can determine that the two NARs 
should be merged or aggregated. Thus, the correlation process may discard the new NAR, replace an older NAR with 
the. new NAR or mark the two matched NARs as candidates for aggregation, a process which is described in detail 

10 below. 

[0109] As part of the correlation process, the flow data collector may enhance 340 the new NAR. That is, the FDC 
may determine that the NAR cannot be correlated without some amount of enhancement. The FDC 52 enhances the 
NAR by supplementing the information provided by the original source equipment with information that is not available 
from that source equipment. The supplemental information is added to the ACCT_ENTITY_ID. Recall that the account- 

75 ing entity identifier ACCT_E NTI TY J D is a collection of descriptors, so the enhancement process 310 adds to that 
collection of descriptors. For example, the accounting entity ID ACCT_E NTITYJ D in one NAR might include a source 
address and a destination address, along with a value indicating how long the flow (for the accounting entity) has been 
in existence. A subsequently processed NAR record having those same three objects can be correlated. However, if 
a subsequently processed NAR only has two of the three objects, the flow data collector can enhance the accounting 

20 entity ID ACCT_E^TITY_ID for the third (missing) object to permit correlation. Enhancement may involve collecting 
information from a completely different network device (via a NAR generated by another accounting process compo- 
nent, such as another data collector), or it may be as simple as adding a timestamp to a NAR's accounting entity ID. 
[0110] As indicated above, the correlation process may determine 342 that two NARs should be "aggregated". Ag- 
gregation merges the accounting entity identifiers of the two NARs together. It also merges metrics for NARs that 

25 contain metrics, as later described. Aggregation of the accounting entity identifiers is accomplished through an explicit 
and implicit matching of those accounting entity identifiers. Correlation relies on the explicitly matched fields, that is, 
the fields or objects actually used to determine that two NARs should be aggregated. The other descriptors or objects 
in the accounting entity ID that were not used by the correlation process to make a match may be equal or different. 
Aggregation of the accountable entity I D portion of the NAR keeps the explicitly matched objects, and determines which 

30 of the implicitly matched objects (the matching objects that were not a part of the explicit match) to save or discard. Of 
course, the nonmatching objects are automatically discarded, as all of the metrics that are the result of this aggregation 
have to apply to the objects in the aggregated accountable entity ID ACCT_ENTITYJD. The removal of accounting 
entity ID descriptors actually serves to lower the semantic complexity of the NAR, whereas enhancement does just 
the opposite. 

35 [01 1 1 ] When the data collection process 330 involves a decision concerning aggregation, the flow data collector 52 
applies 344 the aggregation policy 322 (from FIG. 14) and uses a method defined therein. The method outlines the 
decision-making process to be followed by the FDC in the case of implicitly matched objects. The aggregation policy 
will be discussed in further'detail with reference to FIG. 18. Once the flow data collector aggregates the accounting 
entity ID ACCT_ENTITY_ID portion of the NAR attributes, it can aggregate the NAR metrics. To aggregate the metrics, 

40 the flow data collector performs a summation process on numerical metric values and/or a logical operation (e.g, 
ANDing ORing, or XORing) on logical metric values. Aggregation of the metrics is specific to each metric field in the 
NAR. 

[0112] Once the NAR aggregation is complete 346, the FDC changes the NAR header (i.e., the NAR_SRCJD and 
NAR_SRC_TIME in the NARJD) of the newly aggregated NAR to identity the component (in this case, the FDC) that 
-tf performed the aggregation as the originator of this particular NAR. The FDC stores aggregated NARs for a period of 
time determined by the configuration profile's event -based counter or timer 320 (from FIG. 14). When the timer expires 
348, the FDC is ready to transfer NARs processed by the correlator/(enhancement) and possibly the aggregator as 
well to the FAR 

[01 1 3] Prior to commencing transfer, the flow data collector 52 determines 350 if the flow aggregation processor 60 
50 is available to receive NARs. If the flow aggregation processor 60 is unavailable, the flow data collector stores 352 the 
NARs to be transferred in its local store 314 (FIG. 1 6). The flow data collector 52 continues to check 354 the availability 
of the flow aggregation processor at periodic intervals until the connection between the flow aggregation processor 60 
and the flow data coWector rs re-estabRshed: When the periodic status check indicates 350 that the flow aggregation 
processor is available, the flow data collector loads 356 NARs into the flow aggregation processor 60. The loading 
55 function can be implemented according to one of many strategies, e.g., a database, file, or data streaming strategy. 
Other strategies could be used. When the flow data collector receives 358 a confirmation or acknowledgment back 
from the flow aggregation processor that the NARs were loaded, the transfer is deemed successful and the locally 
stored copies of the transferred NARs are removed 360 from the local store. Thus, the "store and forward" capabilities 
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of the flow data collector provide a measure of fault tolerance at this accounting process level to ensure reliable data 
transfer. The flow data collector only transfers NARs when it has determined that the flow aggregation processor is 
available and it cdnsiders the NAR transfer successful only upon receipt of an acknowledgment from the flow aggre- 
gation processor. 

s [0114] The flow aggregation processor (FAP) 60 (FIG. 2) aggregates and/or enhances record data across the system 
10. It receives data from multiple flow data collectors (FDCs) that may be aggregating and enhancing close to the 
source of the information (as described above with reference to FIG. 17). As NARs are received from multiple FDCs, 
the data can be further enhanced and/or reduced (i.e. aggregated) to meet the specific needs of an application or 
output interface based on the aggregation policy of the flow data processor 60 (FAP). The design and operation of the 

10 FAP will be described in more detail below. 

FLOW AGGREGATION PROCESSOR 

[0115] Referring now to FIG. 16, one implementation of the FAP 60 is as a database management system, or more 
75 specifically, a Structured Query Language (SQL) database management system/like those commercially available 
from Oracle or Sybase. Although not shown, it will be appreciated that the FAP is installed on a computer system, such 
as a host computer. Implemented as a database management system, the FAP includes a database server 400 coupled 
to a database 402. The FDCs 52 (from FIG. 14) can use the "push" model to move NARs up to the FAP via SQL calls. 
The database 402 stores a plurality of tables 404, including a NAR table 406 (implemented as a persistent cache) and 
20 an aggregation store 408. Also stored in the database are a plurality of SQL commands and procedures (functions) 
410 to be executed by the server 400. The functions include a FAP correlator 412, a FAP enhancer (enhancement 
process) 414 and a FAP aggregator 416. The database also stores a configuration file 420 for storing configuration 
parameters such as time and policy information. The operation of the FAP will be described below with reference to 
FIG. 17. 

. 25 [0116] Referring to FIG. 17, an overall flow aggregation process 430 performed by the FAP is shown. The FAP 
receives 432 a NAR from one or more FDCs and loads 434 the received NAR into a persistent store or cache (of 
database 492 from FIG. 16). If the FAP is unable to load the NAR, it requests 436 that the transferring FDC resend 
the NAR. If the load is successful, the FAP sends 438 an acknowledgment back to the sending FDC. The FAP deter- 
mines 440 if the NAR can be correlated (with or without enhancement). If the FAP determines that the NAR can be 

30 correlated, the FAP correlates 442 the NAR with other NARs received from other FDCs. Once the NAR is correlated, 
it may be enhanced 444 "across the system", in a manner more fully described with reference' to FIG. 18. The NAR 
may be enhanced 446 to include enhancement information obtained from an outside source (i.e., collected by a data 
collector for a different equipment interface). Once any potential correlation and enhancement has been performed, 
the FAP determines 448 if the NAR is a candidate for aggregation. If so, the FAP applies 450 the aggregation policy 

35 420 (from FIG. 16) and stores 452 the resulting aggregated NAR in the aggregation store until a predetermined time 
expires or event occurs 454 (as set in the FAP configuration 420). The FAP ensures 456 the uniqueness and integrity 
of any NAR by examining NAR header information prior to re-loading 458 such NAR into the persistent store. 
[0117] The accounting architecture may be implemented to include a second "shadow" FAP process, also coupled 
to the data collectors and operating in the manner described above with respect to receiving and processing NARs. 

40 in the dual/shadowing FAP implementation, the accounting architecture further includes an error detection module (not 
shown) coupled to both of the first (primary) and second (shadow) FAP processes. The error detection module operates 
to detect an error relating to the first flow aggregation process and cause the aggregate reports from the second flow 
aggregation process to be transferred to the accounting module (i.e., flow data distributor 70) in place of the aggregate 
reports from the first flow aggregation process. 

45 

ENHANCEMENT ' 

[0118] Now referring to FIG. 18, an example of an application of the FAP enhancement process 444 (from FIG. 20) 
is shown. In the illustrated example, enhancement deterministically identifies the source of a captured network ac- 
so counting record, flow or a transaction across a network. Enhancement accesses other sources of information on the 
network in order to enhance a record and make it chargeable to a specific user. 

[0119] In the example shown in the figure, two NARs of different sources are inevitably going to be aggregated 
together to produce a third unique NAR. A first source equipment (or source) 500 rs a DHCP (Dynanjrc Host Configu- 
ration Protocol) server. A second source equipment (or source) 502 is a flow probe (discussed below). The sources 
55 500, 502 have corresponding flow data collectors, a first FDC (FDC1 ), 504 and a second FDC (FDC2) 506, respectively, 
for converting their data into respective NARs NAR1 508 and NAR2 510. As described earlier, each flow data collector 
assigns an accounting entity identifier 512,514, and adds time stamp information 51 6, 51 8 on the records of the sources 
to which they correspond. The NAR1 508 includes in its assigned accounting entity identifier 512 an "IP address-to- 
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username" assignment, thus including an IP address 522 and a username 524. The accounting entity identifier 514 
lor the second source is an IP-to-IP flow and therefore includes a first IP address 526 and a second IP address 528. 
The NAR2 of the flow'probe includes a metric 530 attribute as well. 

[01 20] These two records NAR1 , NAR2 are combined through correlation 442 (from FIG 1 7) and enhancement 444 

5 (FIG. 17) to generate an enhanced NAR2 532. This enhanced NAR has a modified accountable entity identifier 534 
and a metric. The modified accountable entity identifier is the existing accounting entity ID 514, to which the FAP has 
added the IP-to-user name assignment 512 from the accounting entity ID 512 of the NAR1 508. 
[0121] Still referring to FIG. 18, the NAR1 508 has an IP-to-usemame mapping 512 and an accounting interval 516 
comprising a start time and a session time to indicate a time interval bounded by start time TV and a start time + 

10 session time ('T2'), that is, the accounting interval represents a start time and a stop time. The username 524 in the 
IP address-to-username mapping is supplied by the DHCP server 500. In the FAR this NAR1 information will either 
go directly to a correlation function or to the local store (which could either be a database, file or memory), where it 
can be directly accessed by the correlator function. The NAR2 510 has an accounting entity ID 514, a T3-to-T4 ac- 
counting time interval 518 and a metric 530. The accounting entity identifier 514 has two IP addresses 526, 528. one 

is corresponding to a source IP address and the other corresponding to a destination IP address. The NAR2 502 is 
passed to the correlator 442, which determines that the T1 -to-T2 time interval 516 from the IP-to-username address 
map in the NAR1 508 overlaps or in some way relates to the T3-to-T4 time interval 518 of the NAR2 510. The correlator 
determines that Tl , T2, T3 and T4 are related, and that the IP address 522 in the IP-to-usemame address mapping 
512 is associated with one of the two IP addresses 526, 528 in the NAR2 510. Thus, the FAP enhances the NAR2 510 

20 by inserting information from the accounting entity ID 512 (of NAR1 508) into the accounting entity ID portion of the 
NAR2 510. The resulting, enhanced NAR2 532 has an enhanced accounting entity ID 534 that includes the T3-to-T4 
timestamp (not shown), the IP-to-IP addresses 526-528 and the username 524. Thus, the enhanced NAR2 now has 
a mapping between the username and the one of the IP addresses 526, 528 that is related to the IP address 522. The 
metric 530 is unchanged. 

2S' [0122] It should be noted that the correlator is able to determine that the time intervals are related to each other 
because the flow data collectors are time synchronized (or closely synchronized, assuming some amount of drift). 
Thus, if the correlator assumes no drift, then T3-to-T4 must be within the time period of T1 -to-T2. The IP-to-username 
address mapping is an event that has to encompass or cover all of the accounting records that apply to that IP address. 
Any user assigned to this IP address, started at T1 and ended at T2. Only those records that reference that IP address 

30 between T1 and T2 will have this username applied to it. When the two flow data collectors are not strictly synchronized; 
then the amount by which T3-to-T4 overlaps T1 -to-T2 should correspond to the amount of tolerance, i.e., drift, built 
into the system. The accounting process assumes a drift amount of at' least one second for even a strict time synchro- 
nization, so T4 can be greater than T2 by one second. Referring now to FIG. 1 9, an aggregation of the enhanced N AR2 
532 (from FIG. 18) is shown. In this example, the aggregation involves combining NARs with IP-to-usemame address 

35 mappings to workgroups. To accomplish this requires two enhancements, two correlations, and an aggregation phase. 
As already described above, with reference to FIG. 1 9, the IP address-to-username information is received by the FAP 
and is either, passed to the correlation or stored in the local store but available to the correlator When' the IP-to-IP 
address NAR with metrics is received, the correlator and the enhancer work together to add the username mappings 
to these IP-to-IP address NAR. The username could be provided for one or both of the source and the destination 

40 addresses. More than likely, the username is assigned to the source IP address. 
• [01 23] Referring again to FIG. 1 9, another correlation and enhancement process 442, 444 maps the username 524 
to a workgroup. The FAP builds up search keys using database principles and relational algebra Thus, for example, 
the IP address has a one-to-one mapping with a username. (The one-to-one mapping is assured because of the nature 
of IP addressing and the way that the DHCP server assigns usernames.) Therefore, there can be only one user for an 

45 ip address in a given instance. These terms or values are equivalent keys : so the username can easily be replaced 
with the IP address. The username 524 that was inserted into the enhanced NAR2 532 can be used as a look-up into 
a workgroup 540 in one of the database tables 404 (FIG. 16) because the user is actually a member ol a workgroup. 
Therefore, the enhancement function can be used to insert the workgroup label into the enhanced NAR2 (already 
enhanced lor username) to produce a twice-enhanced NAR2 542. If the now twice-enhanced record 542 is to be 

so aggregated, it is held in the aggregation store 408 (FIG. 1 6) for some time period T until other NARs are received for 
potential aggregation. 

[0124] Suppose nowthat another NAR is loaded into the FAP. This new NAR passes to correlation, which determines 
that enhancement is need in order to correlate the new NAR with the twice enhanced NAR2 542 of Fid. T9. As a resutt, 
the FAP enhances the NAR to include the username 524 and the workgroup 540 to produce a resultant NAR "NAR3" 
55 550, as shown. 

[0125] Referring to FIG. 20, in addition to the username and the workgroup, the other NAR3 attributes include the 
T3-T4 accounting interval, the IP-IP address mapping and the metrics. With the enhancement, the correlation process 
444 determines that the resultant NAR3 now matches the twice enhanced NAR2 542 held in the aggregation store 
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408. Having explicitly matched the two NARs, aggregation 448 is performed. Aggregation preserves. the explicitly 
matched data objects that are in the accounting entity identifier, discards any mismatches in the accounting entity 
identifier and makes a decision whether to keep the implicitly matched objects (i.e., those that seem to be equal but 
were not used to make the correlation match). It also then combines the relevant metric values together via summation 

5 or logical operations (e.g., ORing, XORing, ANDing). Once the aggregation is complete, the FAP holds the resulting 
aggregated N AR 552. As the FAP receives additional NARs, the aggregator continues to sum and perform these logical 
operations on these metrics values for some aggregation period. The duration of that aggregation period, may be in 
the order of 60 seconds to a week, or however long the FAP is configured to aggregate these records. The termination 
of that period can be a time-based or event-based. Once an event that terminates the time period occurs or an aggre- 

10 gation timer expires, the aggregated NARs held in the aggregation store are released for output by the FAR 

AGGREGATION ADJUSTMENT 

[01 26] It can be understood from the foregoing description that aggregation exists at different levels of the accounting 
15 process. As shown and described above with reference to FIGS. 15 and 17, both the flow data collector and the FAP 
are aggregation-capable. Each aggregates in accordance with an overall aggregation policy that defines how aggre- 
gation is used to provide the data to meet the needs of a specific application. The aggregation performed by the different 
levels can also be remotely and independently adjusted, as will now be described. 

[0127] Aggregation adjustment involves the ability to adjust the level of aggregation to meet specific application data 
20 needs. There are two aspects of aggregation adjustment: remote control and variable degree. ' 

[0128] Referring to FIG. 21 , a graphical representation of aggregation control and adjustment via a data flow decom- 
position model is depicted. As shown, the accounting system is depicted as a tree 560. The flow data collectors are 
leaf nodes 562a-562e and the two illustrated FAP processes are intermediate nodes 564a-564b. The root 566 is the 
collective view of all of the processed accounting information. Given a common view of all the data and the particular 
25 accounting information requirements of a given application, the root 566 thus embodies a single accounting/aggregation 
policy 568. The accounting policy is defined such that an accounting schema is a direct derivative of the accounting 
policy 568. 

[01 29] The accounting policy 568 is viewed as a collection of accounting objects 570, each defined as an accounting 
entity identifier 572 and a set of metrics (not shown). The accounting entity identifier is an abstract object resulting 
30 from construction functions that use the flow data collector data as its original starting point. If an accounting entity ID 
is in the accounting policy as a part of a collection of accounting objects, it is there because it can be constructed from 
the FDC data and the collective set of operations that allow for correlation, enhancement and aggregation. Therefore, 
if an accounting entity ID can be constructed, it can be decomposed. 

[0130] To implement a given user/application requirement, therefore, the data flow model 566 decomposes each 
35 object's accounting entity ID into policy information 572a-g, which includes a collection of data 574 that can be supplied 
by the available flow data collectors and a set of functions or methods 574 needed to correlate, aggregate or enhance 
that data in order to construct the accounting entity identifier. 

[01 31] Aggregation adjustment takes an accounting policy that is a collection of accounting objects and decomposes 
those accounting objects into their accounting entity identifiers and then further decomposes the accounting entity 

•to identifiers in a recursive fashion to provide the collection of basic data and functions needed to construct those ac- 
counting identifiers. This concept builds on the logical directed graphs as seen in many compilers or data flow systems. 
Knowing the order of the functions, the data requirements and dependencies, the data flow software can build the 
logical graph from the decomposition and that specifies data requirements and methods that can be distributed to 
configuration files in the flow data collectors and FAPs to result in adjusting the configuration of .those accounting 

is components. 

[01 32] For example, suppose a user wants to receive accounting on an hourly basis from all of the potential sources 
of information. The flow data collectors 562a-562e are the components that are available for collecting the raw infor- 
mation to generate the accounting data in accordance with a user-specified accounting policy. The internal FAP proc- 
esses 564a-564b further correlate, enhance and aggregate to evolve the data towards the overall accounting data to 

50 meet the accounting policy 568 specified by a user. Thus, the user's information requirements are translated into a 
policy (i.e., collection of objects), which is received by the accounting system and decomposed into the sets of data 
requirements and methods for each of the available accounting components 562a-562e, 564a-564b, that is, policy 
information 572a-572g). Assuming that these components or processes are already confrgured; these sets represent 
configuration updates that are distributed to and stored in the configuration files (see FAP configuration file 420 from 

55 FIG. 16 and FDC configuration file 318 from FIG. 14) in their respective processes. 

[0133] Referring now to FIG. 22, a depiction of the configuration update is shown. The decomposition/configuration 
update process is implemented in software and is based on known data flow technology used in conjunction with an 
available visualization tool to act as a front-end graphical interface. Using such visualization tools, the updated con- 
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figuration is simply mapped to the appropriate component. 

[01 34] It should.be noted that not all accounting processes have a complete collection of data collectors. For instance, 
if the accounting process is to perform user-based accounting and the accounting process only has a flow probe, then 
it will be necessary to request that the user supply a static table of IP-to-username mappings or a source of DHCP 
5 user IP address mappings. The source of that "outside" information becomes part of the decomposition strategy 

INFORMATION MANAGEMENT 

[01 35] The NAR sequence number (N AR_SEQ_NUM FIG. 8B) allows components that are in the next level to detect 
io if there are missing NARs in a collection of NARs and can be used to give a sense of how often NARs are produced 
in a given time period. With the time stamps and the sequence numbers, a per second creation rate of N ARs throughout 
the system can be determined. With this information being part of every NAR. the accounting process 14 can determine 
a sense of the functional capabilities of the intermediate components and detect some aspects of the communication 
channel between components. Also included in a NAR identifier is a* component type identifier 207a which specifies 
J5 what kind of component produced the NAR and its serial number 207b as described above in FIG. 8B. The component 
type identifier allows the accounting process 1 4 to keep component statistics and characteristics based on component 
type. It also allows specific processing on the NARs. NAR IDs are allocated in a very specific way through a management 
system in order to insure that the IDs are actually unique within the accounting process 14. 

[0136] Referring now to FIG. 23, the sequence numbers (NAR_SEQ_NUM) are a key reliability feature 590 of the 
20 accounting process 14 By having the sequence numbers as part of the NARs and knowing that the numbers are 
monotonically inci easing enables the accounting process 14 to track and identify 592 lost traffic or records. It also 
enables the accounting process to determine 592 the amount of lost traffic. By having the NARs with stored accounting 
process component IDs (e.g. a data collector assigned to a particular network device that is allocated at the time that 
the collector is assigned) the information management process 590, can identify 594 the data collector responsible for 
' 25* the flow. The accounting process 1 4 can call back to the data collector that produced the NARs of a particular flow and 
request 596 that the missing NARs (i.e., those NARs for which there are.missing sequence numbers) be retransmitted. 

FLOW PROBE 

30 [01 37] As discussed above in reference to FIG. 2. the accounting process supports a flow probe e.g., 1 2c that cap- 
tures a user's network activity for purposes of I P accounting. The flow probe 1 2c monitors all traffic over a given network 
link and captures data associated with the different flows in the traffic on that link. It is capable of monitoring IP data 
flows over a number of technologies (e.g., Ethernet, ATM, FDDI, etc.). 

[0138] One important feature of the flow probe is its ability to detect and report on successful and unsuccessful 
35 connectivity. This capability is useful to billing and chargeback applications. For example, a user may try to connect to 
a particular switch or reach a particular network, but is rejected. The flow probe 12c can identify that transaction as 
unsuccessful and provides the billing application with information that the billing application can use in determining 
whether or not the user should be charged for that transaction. The flow-based connectivity model embodied in the 
flow probe is described generally with reference to FIGS. 23-25, and specifically with reference to FIGS. 27-28. 
40 [0139] Referring to FIG 2± a representation of a network 600 in which an end system "A" 602 is connected to 
another end system "B" 604 is shown. The terminal systems A 602 and B 604 communicate with one another over a 
communication path 605 Along that path are multiple intermediate devices 608 (e.g., routers, switches) to support the 
communication services required for communications between A and B. Although the path from A to B is depicted as 
a single straight line, it may be appreciated that the actual physical topology of this path most likely is extremely complex.. 
is For the purpose. of understanding the flow probe's connectivity model, however, it is not necessary to know how the 
actual network would be conligu r cd 

[0140] The physical deployment of the flow probe in a network, such as the network 600, is based on two criteria: 
performance, e : g., a 100 Mb probe must be deployed within a region of the network that operates at 100 Mb, and 
granularity of the information to be generated. That is, if the performance or the quality of service provide by A is of 
so particular interest, then the flow probe is located as close to A as possible so that the flow probe will see all of the traffic 
that is seen by A. 

[0141] The deployment of the flow probe may be in-line or out-of-line of the stream of IP packets, of interest. Thus, 
the flow probe 12 may be deployed in-line, i.e., integrated into either of the components that are actualry party to a 
conversation (like end station A 502. as shown in the figure), one of the devices 608 that are actually supporting the 
55 communication or out-of-line. i e . packets are copied and delivered to a remote position. 

[01 42] Generally, a flow is defined as any communication between communicating entities identified by an IP address, 
a protocol and a service port All IP packets (or datagrams) are categorized using the fields present in the packets 
themselves: source/destination IP addresses, the protocol indicated in the IP header PROTO field, and, in the case of 
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UDP or TCP, by the packet's source and destination port numbers. 

[01 43] In a given network segment monitored by the flow probe, much of the typical IP traffic includes TCP protocol 
traffic. Because the flow probe is a flow based monitor that is actually tracking the TCP as a flow, it is completely aware 
of the TCP protocol and that protocol's three-way handshake algorithm (state machine). The TCP flow has indicators 
to indicate that a connection is being established or a flow is being disconnected. However these messages are only 
relevant to the two communicating parties (e.g., A and B in FIG. 27). The end system A may request that it be able to 
communicate with B and sends a "TCP SYN" indication. Any of the networking devices 608 along the path 606 can 
reject this SYN request, completely independent of the intended destination (in this example, end system B) and without 
the knowledge that the end system B is a party to this communication request. There are a variety of problems that 
can cause an internal network component to reject a request. For example, a router between A and B may find that 
there is no route available for forwarding a packet towards B or that the routing path is inoperable (and no alternate 
exits), or the router may find that it doesnt have the resources to handle the packet. 

[01 44] The Internet Control Message Protocol (ICMP) is designed to convey this type of error event information back 
to the originator of the request. For example, suppose device 608 is a router that is in a "failed" state and cannot 
process the SYN request that it received from A. The support exists in the Internet protocol, specifically, ICMP, to signal 
this condition back to A. Originator A has the ability to correlate the error event with the request and inform the requesting 
application that its request is not going to be supported. Because the network uses a completely independent protocol, 
i. e. ICMP, to convey the information, it is necessary to correlate these independent protocols (TCP and ICMP) to provide 
the accounting process with the information it needs to know about a given transaction. Specifically, the accounting 
process needs to know if the transaction was successful or unsuccessful and the cause of failu're if unsuccessful. 
[0145] As an independent monitor operating outside of the context of the originating entity ("A",, in this example), the 
flow probe is able to produce a complete and accurate record of the transaction by mapping the network control infor- 
mation to the user request information. To do so, flow probe correlates the state information in protocols such as TCP 
with error event or condition messages provided by other protocols, such as ICMP. In this manner, it is possible to 
determine if a particular request for a service has actually been denied as a result of some network independent event. 
The flow probe correlates the dissimilar protocols together and finds a way of representing the network event in its 
normal reporting of the TCP flow. 

[0146] The flow probe has specific reporting mechanisms for the specific protocols. The TCP protocol, for instance, 
has many more metrics associated with its protocol states than UDP based flow's. However, because ICMP relevant 
events or network relevant events are not associated with or have any impact on the state of TCP or UDP or any of 
the normal protocols, the flow probe provides a mechanism f or tagging its state tracking with the error event. The NAR 
is represented as a start flow indication, a continuing or status record and a stop record. All of the flow probe's internal 
protocol indications map to, start, continuous or stop states. When a network rejection event comes in (e.g., in the form 
of an ICMP message, or other type of internet control information), regardless of what state the probe is tracking as 
the current state, it reverts to a stop state and has to expand upon the normal time or transition based stop conditions 
to include an specific ICMP event as the cause of the closed state. The flow probe NAR includes bit indications for the 
actual protocol states that it is tracking. For ICMP generated events, the flow probQ indicates whether the source or 
the destination was affected by the events. In order to convey this network rejection or network event back to the parent 
flow, the NAR allows for specific network rejection logic to be reported either by the source or the destination, and has 
specific bit indicators in either the source or the destination fields. 

[0147] There are two key aspects to the connectivity scheme of the flow probe as described thus far. First, the probe 
determines that an ICMP event has occurred. Second, the probe correlates that event to the "parent" flow, i.e., the 
same flow as that associated with the failed request, and stores the exact ICMP event into some state associated with 
that flow so the event can be reported to the accounting system in a NAR. At this point it may be useful to examine 
the IP packet and ICMP message formats in general, as well as examine certain fields of interest. 
[0148] Referring to FIG. 25 : an exemplary IP packet format 610 is shown. The IP packet format 610 includes an IP 
packet header 612 and an IP packet data field 614. The IP packet header 612 includes a PROTOCOL field 616 for 
indicating the protocol of the message encapsulated therein. The header also includes a source IP address field 618, 
a destination IP address field 620 and other known fields (not shown). In the example of FIG. 25, the message contained 
in the IP packet data field (or payload) is an ICMP message or packet 622. The ICMP packet is formatted to include 
an ICMP header 624 and an ICMP data field 626. In the example, the protocol field 616 would be set to indicate a 
protocol value corresponding to ICMP. 

[0t49] Referring to FIG. 26, an exemplary ICMP message format 622 for reporting errors ts shown: The format 
includes an ICMP message header 624. The header 624 includes a type field 630, which defines the meaning of the 
message as well as its format, and a code field 632 that further defines the message (error event). The error reporting 
message types (type values) include: destination unreachable (3); source quench (4); source route failed (5); network 
unreachable for type of service (11); and parameters problem (12). Each of the types has a number of code values. 
For a destination unreachable message (TYPE field value is 3), the possible codes (code values) include: network 
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unreachable (0); host unreachable (1); protocol unreachable (2); port unreachable (3); fragmentation needed and DF 
set (4); source route failed (5); destination network unknown (6); destination host unknown (7); source host isolated 
(8); communication with destination network administratively prohibited (9); communication with destination host ad- 
ministratively prohibited (1 0); network unreachable for type of service (11 ); and host unreachable for type of service (1 2). 

s [0150] Also included in the ICMP message format is a datagram prefix field 634, which contains a prefix -- header 
and first 64 bits of data -- of the IP datagram that was dropped, that is, the datagram that triggered the error event 
message. The datagram prefix field 634 corresponds to the ICMP message (packet) payload: The IP datagram or 
packet header 612, partially illustrate in FIG. 24, is shown here in its entirety. Assuming that the IP datagram carries 
a TCP message, the protocol value would correspond to TCP and the portion of the IP datagram's data 636 (first 

10 64-bits) would in fact correspond to a TCP message header 636, which includes a source port field 638, destination 
port field 640 and a sequence number field 642. The source port is the port number assigned to the process in originating 
(source) system. The destination port is the port number assigned to the process in the destination system. 
[0151] It will be understood that TCP is an example protocol. The field 636 could correspond to a portion of packet 
header from a packet of another protocol type. Also, the error reporting protocol could be a protocol other than ICMP, 

is and the amount of header in field 636 could be more or less than 64 bits, that is, this amount may be adjusted so that 
the appropriate flow information can be obtained from the header ol the message contained in the discarded IP packet, 
as described below. . 

[0152] Referring to FIG. 27, a packet processing method ("the process") 650 performed by the flow probe is shown. 
The process captures 652 a new IP packet (datagram) and tests 654 the received packet todetermine if it is good (i. 

20 . e ., well-formed). T ( he process 650 examines 656 the protocol field in the IP packet header to determine if the protocol 
is the ICMP protocol. If the protocol is ICMP and the information type field is set to one of the five error reporting 
messages described above, the process bypasses the IP packet and ICMP message headers and processes 658 the 
ICMP message or packet payload (FIG. 26), which corresponds to a portion of IP packet which that was discarded 
and to which the event message relates. The payload process will be described with reference to FIG. 28 below. Once 

25 the payload processing is complete, the processing of the IP packet resumes 659 the processing that would be per- 
* formed if the IP packet had not been detected as containing an ICMP message of the error reporting variety as discussed 
above, as will now be described. 

[0153] Still referring to FIG. 27, if the protocol is not ICMP and/or the information type is not an error report, the IP 
packet is processed as follows. The probe scans 660 the header to determine the values of the fields which correspond 

30 to the "flow key", the fields which define "the flow" for the probe. Each flow probe can be configured for a particular 
flow key definition. For example, the flow key might be the source/destination IP addresses, the source/destination 
ports and the protocol. The probe determines 662 if the flow key of the processed packet header matches a flow already 
stored in the flow probe. A local store in the flow probe is used to hold flow representations including flow key param- 
eters, metrics, state information. The state information will include, in addition to the protocol control-related states (i. 

35 e., TCP "FIN"), error event/state change cause and source/destination to which the message is addressed. These flow 
representations are converted into NARs for accounting process reporting purposes. 

[01 54] Still referring to FIG. 27, if the flow probe cannot match 664 the flow key information to a stored flow, the probe 
constructs (and stores) 666 a new flow and completes 668 the process. If the probe finds a match, it updates 670 
metrics for the matching stored flow (or ■parent" flow). It also updates 672 the flow state of the parent and then completes 
40 674 the process. It should be noled that the construction of a new flow triggers 676 the generation of a start NAR and 
the update of the flow state triggers 678 the generation of an update NAR. The generation of NARs by the flow probe 
will be discussed later. 

[0155] Referring to FIG. 28, processing of the ICMP message payload, i.e., the embedded IP packet, 658 (.from RIG. 
27) is shown. The processing of the ICMP message payload processing is recursive in nature. The essential method 

45 is the same as used above for an IP packet (FIG. 27), with a few differences. 

[01 56] If the flow probe determines 664 that there is no stored flow corresponding to the flow of the dropped IP packet 
or datagram (indicated by the ICMP message in the data prefix field or payload 634 of FIG. 26), the processing is 
complete 630. If a stored flow matching the flow key of the dropped datagram is found, the probe updales 672 the flow 
state to indicate a "rejected" state for the stored flow. It also updates the flow state information to indicate whether it 

so was the stored flow's source or destination that was associated with the ICMP message and the event cause. The 
state change (to rejected state) triggers 632 the generation of a stop NAR, as is later described. Once the probe has 
completed the payload processing 658, it resumes 659 the processing of the original IP packet (as indicated in FIG. 27). 
[OT57J Thus, the payload processing can be viewed as a packet processing exception, an exception that rs invoked 
when it is determined that an ICMP error reporting message has been received. The ICMP message reports a error 

55 event and the IP packet associated with that error event. The exception process serves to correlate the flow of the 
discarded IP packet in the ICMP message with the parent (matching stored) flow, thus mapping the ICMP error (state) 
information to the parent IP flow. 

[01 58] The flow probe reports on network traffic activity through a flow probe NAR, which reports IP flow traffic activity. 
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The flow probe categorizes network traffic into one of four classes of traffic flow: I) connection oriented (e.g., TCP); ii) 
new connectionless; iii) request/response connectionless (e.g., UDR DNS); and iii) connectionless persistent (e.g., 
NFS, Multicast BackBONE or "MBONE" multicast traffic). To each of these class it applies connection oriented seman- 
tics for a uniform approach to status reporting. That is, flow probe treats these dissimilar transaction models as if they 

5 were the same. There is one uniform structure for the status reports generated for each of the 4 different transactions. 
Each status report includes transaction start and stop information, MAC and IP source and destination addresses, the 
I P options that were seen, the upper layer protocol used, the transaction source and destination byte and packet counts 
and upper layer protocol specific information. The protocol specific information and the criteria for when the status 
reports are created, is different for each of the four transaction types. 

w [0159] The connection oriented protocol understood by the flow probe is TCP Flow probe has complete knowledge 
of the TCP state machine and thus can generate status reports with each state transition seen within any individual 
TCP. There is also a provision for generating time interval based status reporting in the TCP connections that the flow 
probe is tracking. The status report indicates which states were seen, if any packets were retransmitted, if the source 
or destination had closed, and if the sport had been generated by a time condition. In a default mode, the flow probe 

'5 generates a cumulative status at the time a TCP closes, or times out. This strategy offers the greatest amount of data 
reduction on transactions. 

[0160] Any non-TCP traffic is categorized as a connection-less transaction. When configured to generate the most 
detailed level of reporting for connectionless traffic, the flow probe can report the discovery of a new connection-less 
transaction: the existence of a request/response pair within the transaction (as exists when the probe has seen a single 
20 packet from both the source and the destination for the transaction); the continuation or transaction persistence, and 
so forth. The transaction persistence status is generated with a timer function. If it has been seen within a configured 
timer window, a report is generated. 

[0161] The status report for non-TCP traffic indicates if the report is an initial report, a request/status report or a 
continuation (or a current transaction) report. 
2S [0162] In the default mode, the flow probe generates a status report when it has seen a request/response "volley ■ 
within a transaction and every 15 minutes thereafter, if the transaction persists. This offer immediate notification of 
request/response traffic and a fair amount of data reduction on connection-less transactions. 

[0163] Thus, the flow probe state tracking includes protocol-specific state information. It provides detailed information 
on transport specific flow initiation such as TCP connection establishment, as well as flow continuation and termination 
30 event reporting. 

PROTOCOL INDEPENDENT PACKET MONITOR 

[0164] Referring to FIG 29A a network 700 includes a monitor 702 that runs a process for detecting packet loss. 

35 The monitor 702 will be particularly described using IP SEC authentication headers. The monitor 702 uses sequence 
numbers that exist in IP SEC authentication headers. The monitor 702 can be used to detect tost packets in any type 
of protocol that uses sequence numbers in headers of the packets, etc. The monitor 702 is an independent monitor 
tha.t can be disposed anywhere in the network 700. The monitor 702 is protocol independent. 
[0165] The network 700 would include a plurality of such independent monitors 702 each disposed at corresponding 

•*o single points in the network 70 Typically, the monitor 702 can be disposed in-line such as in a network device such 
as a switch, router, access concentrator, and so forth. Alternatively, the monitor can be disposed in an out of line 
arrangement in which network packets are copied from the device and coupled to the out-of line monitor. 
[0166] The monitor 702 examines each packet of a network flow that passes through the device associated with the 
monitor 702. The monitor 702 receives serialized I P packets. The packets can have the format specified by the Network 
Working Group, by S. Kent Request for Comments: 2402, November 1998 "IP Authentication Header" as part of the 
"Internet Official Protocol Standards', The Internet Society (1 998). The IP Authentication header includes a Next Header 
field that identifies the type ot the next payload after the Authentication Header, Payload Length an 8-bit field specifies 
the length of AH, and a reserved 16-bil field. The IP Authentication header also includes a Security Parameters Index 
an arbitrary 32-bit value that in combination with a destination IP address and security protocol, uniquely identifies the 

so Security Association for a datagram and a Sequence Number. The sequence number is an unsigned 32-bit field con- 
taining a monotonically increasing counter value (sequence number). It is always present in such datagrams and is 
provided form the purpose to enable an anti-replay service for a specific security authentication. According to the 
standard if anti-replay is enabled the transmitted Sequence Number rs not attowed to cycle. Thus, the sender's counter 
and the receiver's counter are reset by establishing a new security authentication and thus a new key prior to the 

55 transmission of the 2 32nd packet. The datagram also includes Authentication Data, i.e., a variable-length field that 
contains the Integrity Check Value (ICV) for the datagram. 

[0167] Referring now to FIG 29B, a packet loss detector process 704 that runs in the monitor 702 is shown. The 
packet loss detector process 704 examines 706 header information in the packet, to determine if the packet includes 
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an authentication header. If the packet does not include an authentication header, then the packet loss detector process 
704 ignores 24 the packet and exits to wait for the next packet. If the packet includes an authentication header, the 
packet loss detectorp'rocess 20 tests 708 to determine if the packet loss detector process 20 had been tracking the 
flow thai is represented by the source and destination IP addresses and the SPID value that is in the authentication 
5 header The packet loss detector will perform a cache look up to determine it the flow is stored in a cache of currently 
tracked flows The packet loss detector process 20 tests 708 those values to see if the packet loss detector process 
704 is cjrrerMly tracking that security flow. 

[01 68] H :hc p^ckci loss detector process 704 is not tracking that security flow, the packet loss detector process 20 
will estrohsh 710^ flow cache entry for that flow in a cache that can be maintained in memory (not shown). The packet 

io lossdciccior process 704 will store the source and destination IP address and the SPID value from of the authentication 
header Tro Ho* c^chc also includes all other authentication headers from other security flows that have previously 
been tracked The flow znchc enables the packet loss detector process 20 to monitor and track many hundreds, thou- 
sands and sc lorn o! different security flows. A cache entry is established for every different flow. Once the cache 
entry is established the packci loss detector process 704 updates 712 the sequence number entry in the cache for 

is that sccjnty How That s ihc initial sequence number in the authentication header for the encountered flow is stored. 
The sequence nunber can sHrt at any arbitrary value. 

[0169] It however the pneket loss detector process 704 determined 708 that it is tracking the flow, then the packet . 

loss detector process 704 icsis 714 if the sequence number in the current packet is equal to the previous sequence 

number no:cd tor this flow pus 1 If the sequence number in the current packet is equal to the previous sequence 
20 number plus 1 then the packet loss deleclor process 704 can stop the current evaluation because the packet loss 

detectoi piucebb 704 did nut delect and the system did not experience any packet loss on that particular association. 

The packet loss deleclor process 704 will update 712 the stored sequence number for that flow in the cache. 

[0170] If the sequence number in the current packet does not equal the previous sequence number noted for this 

flow plus 1 . the packet loss detector process 704 for the IP SEC Authentication packets detected a potentially missed 
25 * packet. 

[0171] For some protocols l hat permit wrap around, the packet loss detector process 704 tests 718 if the sequence 
number, has wrapped around e g.. gone from 32 bits of all ones to 32 bits of all zeros. The IP SEC Authentication 
packets currently do not permit wrap around, so test 718 would not be necessary for IP SEC Authentication Headers. 
If for other protocols (or lallor versions of the IP SEC Authentication protocol), the packet loss detector process 704 

30 detects a wrap around condiion then there has not been any packet loss and the packet is dropped. The packet loss 
detector process 704 will update 712 the stored sequence number for that flow in the cache. If the sequence number 
is any other number, i.e . it did not turn over to all zeros, then there may have been packet loss. If there may have been 
packet loss, the packet loss detector process 704 can determine how many packets have been lost by determining 
how many sequence numbers are missing. 

35 [0172] When packets may traverse more than one packet monitor 10, the packet loss detector process 704 may 
produce a packet loss detected indication that does not indicate that the packets were actually dropped. A packet loss 
drop indication in a multi-monitor embodiment indicates that the lost packets did not come through the particular packet 
loss detector process 704 However, the indicated lost packets could be on other segments of the network. That is, it 
is possible that other parts of the current flow are in other parts of the network. Therefore, the packet loss' detector 

40 process 704 notes how many packets were actually successfully transmitted, as well as lost, and optionally their se- 
quence numbers. These values can be compared to other values from other monitors 702 to establish whether or not 
there had been packet loss for the flow through the network 

[0173] This indication, could be converted into Network Accounting Records thus would be coupled to a process e. 
g. the accounting process 14 that reports statistics on that particular flow to provide a summary of how many packets 
is were lost relative to how many packets were actually successfully transmitted on the flow. In the accounting process 
14, the network accounting records are correlated, aggregated, enhanced and so forth to identify network flows. This 
information can be used to determine the records that correspond to a particular network flow and whether a determined 
nelwork flow lost any packets 

SO CAPTURING QUALITY OF SERVICE . 

[0174] Referring now to FIG. 30. a process 730 for capturing quality of service in a network system 11, (FIG. 1), is 
shown. The capturing quality of service process 730 allows an administrator to configure 732 the network TT with a 
policy that corresponds to a first quality of service. The process 730 also includes an optimization process that assigns 
55 or develops 734 the policy defines the policy being used, and enforces the policy by deploying a policy dictated con- 
figuration into various policy enforcement devices in the network 11 . The capturing quality of service process 730 allows 
the administrator to observe 736 the actual service delivered by the network 11 to a customer on the network 11 to 
determine if the quality of service provided matches that specified by the policy 740. 
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[01 75] The capturing quality of service process 730 uses an accounting process 738 io collect information from the 
network. A preferred accounting process is accounting process.1 4 described above. The accounting process 1 4 collects 
data from. the network 11 as part of the observation process 736. The accounting process collects different kinds of 
metrics from the network, correlates these metrics to specified network flows : via the use of N ARS, and maps collected, 

5 correlated information i.e., NARs back to the policy that was defined and actually deployed in the network. Because 
the accounting process 14 performs this observation function, the accounting process can provide an indication 738a 
whether or not the policy 740 is being satisfied. ' ' 

[0176] By deploying the accounting process 14 to observe service quality, the capturing quality of service process 
730 can validate performance of service level agreements (not shown). If the capturing quality of service process 730 

10 detects that the policy level specified in a service level agreement is not being enforced, then the policy can be reas- 
sessed, redefined, and redeployed 742. The capturing quality of service process 730 can again observe 737. Through 
the observation 736, the capturing quality of service process 730 can determine whether reassessment and redefining 
of the deployed policy was successful. Several cycles of this quality of service optimization process could be required. 
[0177] An important component of quality of service includes determining whether there has been packet loss. The 

is packet detector monitor described in conjunction with FIGS. 29A and 29B can be used to access packet loss. The 
packet detection monitor 702 can be deployed in the network and generate NARs that can be used to determine packet 
loss as discussed above. This information can be used in the capturing quality of service process 730 to assess whether 
the policy specified by the service level agreement was provided to the customer. Additionally, so called Differentiated 
Service "DivServe technology" that a known quality of service solution that has been proposed for the internet as well 

20 as enterprise networks. In contrast to a per-flow orientation of some types of quality of service solutions such as Int- 
serv and RSVR DiffServ enabled networks classify packets into one of a small number of aggregated flows or "classes", 
based on bits set in the type of service (TOS) field of each packet's IP header. This is a quality of service technology 
for IP networking is designed to lower the statistical probability of packet loss of specific flows. The capturing quality 
of service process 730 establishes DivServ policy, that is decomposed into a collection of DivServ configurations. The 

25 DivServ configurations are deployed to a collection of routers or switches that the customer would have access to in 
the network 11 as part of the enforcement/deployment process 732. Because packet loss is a statistical phenomenon, 
the capturing quality of service process 730 observes 736 a large number of network flows. The capturing quality of 
service process 730 can observe network traffic because of the use of the accounting process 14 and the resulting 
NARs at the granularity in which the DivServe policies are actually being deployed. The DivServe policies are generally 

30 deployed at the source and destination IP address, protocpl and possibly destination port level. 

[01 78] By observing 736 network flows at the same granularity as a DivServe policy enforcement mechanism, if the 
capturing quality of service process 730 detects packet loss at that granularity, then there will be a direct feedback 
coupling to determine whether the policy is actually being enforced or not. If the policy is not being enforced, then an 
administrator will can reassess the policy, redefine the policy, and redeploy 742 new enforcement strategies. The 

35 capturing quality of service process 730 again will observe 736. 

[0179] As mentioned, because IP network quality of service is a statistical phenomenon, the capturing quality of 
service process 730 obtains a large number of samples, over a long period of time. Through this optimizing capturing 
quality of service process 730 and DivServe deployment 734, the customer will get beneficial policy deployment for 
this service. 

40 

SERVICE MANAGEMENT ; ' 

[0180] Referring now to FIG. 31 , a service management loop 750 includes a service provisioning application 752, a 
policy enabled network server 754 and. an accounting process 756. In a typical example, an Internet Service Provider ' 

4 $ (ISP) and a customer will enter into a service agreement or contract 751 that will specify a level of service for the 
network. The contract 751 has requirements and conditions that are enforced by the policy enabled network 754. The 
service contract 751 is decomposed by the policy server to produce a template that defines the service represented 
by the agreement 751 . The template is fed to the service provisioning application 752 that actually produces a config- 
uration file 752a that is sent out to the network 10 to configure network for a level of service based upon that contract 751 . 

so [0181] A service management feedback process 750 therefore includes three components, service provisioning 752, 
policy server 754 and service accounting 756. The role of service provisioning 752 is to send requests 752b to the 
policy server 754 to obtain an appropriate active policy, and obtaining rules and domain information 754a from the 
policy server. The provisioning system can communicate with appropriate network management systems and element 
management systems (not shown) to configure the network 10 for an end-to-end service. When the configuration 752a 

5S js deployed at the various network devices (not shown) at that point, the service is produced. The level of service is 
monitored or audited by the accounting system 756 which can be the accounting process 14 described above. The 
accounting process 14 monitors the level of service by producing appropriate network accounting records. The network 
accounting records NARs are used by a billing application to adjust billing based on the level of service that was 
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provided as determined by the accounting system 14. The accounting system 14 also can compare the policies pro- 
duced by the policy server to the actual levels of service provided to the customer by examining NARs that are produced 
by the customer's usage of the network. 

[0182] In addition, levels of service might change, and the system takes changes into account so that the service 
s management can modify the charge or account differently for those changes in levels of service. The service accounting 
also uses the active policy information from the policy server to deliver billing information to a billing system or to a 
chargeback system that can may adjustments to billings for the service. 

[0183] A policy enable network 754 is build on the capabilities of address management, domain name management 
and so forth. Essentially in a policy enabled network, policy services produce a set of rules and applies those rules to 

to a domain or problem set. The policy server communicates the rules to the accounting process 1 4 so that the accounting 
process 14 can determine what kind of records to generate. All of the information is described using data flows. 
[01 84] As an example, a service contract may specify that a company "X" will be given 1 00% availability of a particular 
network device e.g., a router (not shown) and its corresponding service. In order to assure that level of service, the 
policy server 754 sends that requirement in a template to the provisioning service 752 to produce a configuration file 

is 752a to configure the router to give company "X" preferred use of the router. Therefore, every time a packet from 
company - X's" network comes across the router, th.e packet will always be transmitted unless there is something wrong 
with the router. This may occur even if a packet of company "Y" which has a lower service level than company "X" is 
waiting in the router to be transmitted. The packet from company "Y" will wait because company "Y B is not paying for 
the quality of service that company "X" is paying for. 

20 [0185] In that case, the provisioning service configures 752 the policy enforcement mechanism that was put into the 
router in the network. How the policy was defined to the provisioning equipment is that there is a one-toone relationship 
between the policy and what the accounting process 1.4 will monitor in the network. The accounting process 1 4 will be 
aware that company "X" contracted to have 1 00% availability from the router. 

[0186] The accounting process 14 will then take every source of information it has available and will construct an 
25 accounting record that reflects the level of service actually delivered to company ."X." The accounting records produce 

are relative to the two components, i.e., the router and the customer. The accounting process 14 is flexible and can 

generate accounting records of any flow abstraction. In this process 750, the policy server 754 sends a flow based 

policy to the provisioning server 752. The provisioning server 752 uses a flow based policy to configure the network. 

That same flow based policy is passed to the accounting process 14 which can generate network accounting records 
30 NARs having metrics that can be used to match the same level of those flows. The output of the accounting process 

1 4 will determine whether the quality of service, availability, etc. that was contracted for in the contract 751 was provided. 

Therefore the service management process 750 provides the level of service that was delivered at the same semantic 

level as the actual contract. 

[0187] Capturing quality of service as audited by the accounting process 14 includes detecting of packet loss, as 

35 mentioned above. Each of the components managed by the service management process 750 require information. 
Therefore, the service provisioning has to provision these various quality levels. The policy server 754 thus, keeps 
what is essentially enforcement of the levels of quality that are offered by different service types, and the accounting 
process 756 detects, monitors and audits whether those classes in quality of service are being delivered. 
[0188] Referring to FIG. 32. an implementation of the service management provisioning 752 is shown. The service • 

40 management provisioning 752 extends concepts of device management and network management into a service man- 
agement layer of functionality Service management provisioning includes a provisioning core 782, provisioning mod- 
ules 784, and element managers 786. Service provisioning 752 is user focused rather that network focused as con- 
ventional network management. Network management involves communication with network systems and equipment. 
Service provisioning 752 is orient more towards a user and a user's concepts of services. Service provisioning 752 

45 provides an additional layer of abstraction that relates description of services at a user level to a network's ability to 
provide those end-to-end services. The architecture 780 of Service provisioning 752 is multi-device 788 at the bottom 
of the architecture and multi-service 790 at the top of the architecture. The service provisioning 752 is deployed to 
write commands lo the network systems i.e., network elements 788 in order to change configurations of those systems. 
[01 89] Since many end customer services now require that a network operate with multiple, different kinds of network 

so elements in order to provide an end-to-end service : the service provisioning 752 simplifies producing information that 
is necessary for a service provider to translate a service order from a customer to a network configuration, i.e., all 
commands necessary for all the different elements in the network in order to create an end-to-end service. 
[0T9O7 The service provisioning buifds orr existing systems. That is, in the tower layers thene are existing element 
managers that have a configuration management system to configure at the network layer The service provisioning 

55 adds layering over the conventional network management layer. Service provisioning maps a customer specified end 
to end service to the network elements that are required to produce that end-to-end service. Mapping of a customer's 
service orders into the state of the network can have various pieces of workflow necessary to create or completely 
activate this service order. 



10 



15 



20 



25 



EP 1 039 687 A2 

. u.' 

[01 91] In summary, the present invention concerns a system for collecting and aggregating data from network entities 
for a data consuming application, is described. The system includes a data collector layer to receive network flow 
information from the network entities and to produce records based on the information. The system also includes a 
flow aggregation layer fedfrom the data collection layer and coupled to a storage device. The flow aggregation layer 
receiving records produced by the data collector layer and aggregates received records. The system can also, include 
an equipment interlace layer coupled to the data collector layer and a distribution layer to obtain selected information 
stored m the storage device and to distribute the select information to a requesting, data consuming application. 

Other EmbxJrncnis 

[0192] l! s lo be understood that while the invention has been described in conjunction with the detailed description 
thereof the lorcgomg ccscnption is intended to illustrate and not limit the scope of the invention, which is defined by 
the scope of the rtppendec clnims. Other aspects, advantages, and modifications are within the scope of the following 
claims 



Claims 

1. A computer implemented method for managing network service comprising: 
iccciving from ri bubbcnbei, a request for a selected network service, 

determining from the request for a selected service configuration characteristics for resources, 
configuring the resources with configuration characteristics to provide the selected service; and 
monitoring the service provided, 

2. A method as claimed in claim i , further comprising: 

billing the subscriber based on the provided service rather than the requested service. 

3. A method as claimed in claim tor claim 2, wherein monitoring further comprises: 

30 collecting data from the network and producing a flow description of the subscribers use of the network. 

4. A method as claimed in any preceding claim, wherein the request is provided by a service agreement. 

5. A method as claimed in any preceding claim, wherein configuring comprises: 

3$ provisioning a service to send requests to obtain an appropriate active policy, rules and domain information. 

6. A method as claimed in claim 5, wherein configuring comprises: 

deployed configuration information at the various network devices at point where the service is produced. 

JO 7. a method as claimed in any preceding claim, wherein monitoring comprises: 

using an accounting process to monitor the level of service by producing appropriate network accounting 
records: and 

•* 5 comparing the policies to the actual levels of service provided to the customer by examining network accounting 

records that are produced by the customer's usage of the network 

8. A method as claimed in claim 7 : wherein using the accounting process uses the policy information to determine 
information that can be sent to a billing system or to a chargeback system to adjust billings for the service. 

50 

9. A method as claimed in claim 7 or claim 8 : wherein the policy is a set of rules; and wherein the method further 
comprises: 

using the rules to determine what kind of network accounting records to generate. 

55 1 o. A method as claimed in any one of claims 7 to 9, wherein the information produced by using the accounting process 
is described using data flows. 



11. A system comprising: 
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a policy enabled network server that can decompose a request for a level of service for the network and 
produce a template that defines the service; 

a service provisioning process to produce a network configuration from the template that defines the service; 
5 and 

an accounting process that monitors the level of service and produces network accounting records. 

1 2. A system as claimed in claim 1 1 , wherein the network accounting records are used by a billing application to adjust 
w billing based on the level of service provided as determined by the accounting process. 

13. A system as claimed in claim 11 or claim 12, wherein the template from the policy server is fed to the service 
provisioning process to produce a configuration file that is sent out to a network to configure network for the level 
of service based upon the request. 

15 

14. A system as claimed in any one of claims 11 to 13, wherein the request is provided by a service agreement. 

15. A system as claimed in claim 1 4, wherein the service provisioning process sends requests to the policy server to 
obtain an appropriate active policy, and obtaining rules and domain information from the policy server 

20 . ( . . 

16. A system as claimed in claim 15, wherein the configuration is deployed at the various network devices at point 
where the service is produced. 

17. A system as claimed in any one of claims 11 to 16, wherein the accounting process monitors the level of service 
25 by producing appropriate network accounting records; and 

compares the policies produced by the policy server to the actual levels of service provided to the customer 
by examining network accounting records that are produced by the customer's usage of the network. 

18. A system as claimed in claim 1 7, wherein the accounting process uses the policy information from the policy server 
30 to deliver billing information to a billing system or to a chargeback system that can may adjustments to billings for 

the service. 

19. A system as claimed in claim 1 2, wherein the policy server produces a set of rules; and 

communicates the rules to the accounting process so that the accounting process can determine what kind 
35 of records to generate. 

20. A system as claimed in claim 1 9, wherein the information produced by the accounting process is described using 
data flows. 

40 . 
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